Introduction
R Packages
Before we begin, we can go ahead and import the required R packages.
If they are not currently installed, that can be done first. The
packages used in this project are: - data.table
- ggplot - plotly
# Install if needed
if (!require(data.table)) install.packages('data.table')
if (!require(ggplot2)) install.packages('ggplot2')
if (!require(ggplot2)) install.packages('ggplot2')
library(data.table)
library(ggplot2)
library(plotly)
Math Fundamentals of the Perceptron Algorithm
In order to understand the Perceptron algorithm, a basic knowledge of
the following mathematical principles must first be understood: -
Vectors (specifically, how to calculate the direction and norm) - The
Dot Product
I am going to assume that majority of people reading this article
have prior knowledge of each of these fundamentals. If you do not, a
great resource can be found here.
I will also give a brief refresher to provide some context.
Vectors
The Dot Product
Linear Separable Data
The definition of linearly separable data is pretty self-explanatory.
At a high level, for data to be linearly separable it simply means that
there exists some plane that can literally “separate” the data. For
example, consider the following plot:

It is pretty obvious that you can draw some arbitrary line that
separates the two different colored data points:

Similarly, consider the following three-dimensional data:
In this instance, a line would be insufficient when attempting to
seperate the two data classes. Instead, a two-dimensional plane can be
used:
Of course, this gets a little harder to visualize - or even explain -
as dimensionality continues to increase. As a result, a more generalized
term to describe this linear separation is used: a
hyperplane.
Hyperplanes
According to the Wikipedia
definition: “In geometry, a hyperplane is a subspace whose dimension
is one less than that of its ambient space.” This makes sense. Looking
at our first example, the deminsionality of the data is two, and the
line drawn to separate the data is one-dimensional. In the second
example, the data is three-dimensional and the separation is two.
Perceptron Algorithm
Generate Data
Weights
Iterate
Multi-Layer Perceptron
LS0tDQp0aXRsZTogIlBlcmNlcHRyb24gTGluZWFyIEFsZ2VicmEiDQpvdXRwdXQ6IGh0bWxfbm90ZWJvb2sNCi0tLQ0KDQojIyBJbnRyb2R1Y3Rpb24NCg0KDQojIyMgUiBQYWNrYWdlcw0KQmVmb3JlIHdlIGJlZ2luLCB3ZSBjYW4gZ28gYWhlYWQgYW5kIGltcG9ydCB0aGUgcmVxdWlyZWQgUiBwYWNrYWdlcy4gSWYgdGhleSBhcmUNCm5vdCBjdXJyZW50bHkgaW5zdGFsbGVkLCB0aGF0IGNhbiBiZSBkb25lIGZpcnN0LiBUaGUgcGFja2FnZXMgdXNlZCBpbiB0aGlzIA0KcHJvamVjdCBhcmU6DQotIFtkYXRhLnRhYmxlXShodHRwczovL2NyYW4uci1wcm9qZWN0Lm9yZy93ZWIvcGFja2FnZXMvZGF0YS50YWJsZS9pbmRleC5odG1sKQ0KLSBbZ2dwbG90XShodHRwczovL2dncGxvdDIudGlkeXZlcnNlLm9yZy8pDQotIFtwbG90bHldKGh0dHBzOi8vcGxvdGx5LmNvbS9yLykNCg0KYGBge3J9DQojIEluc3RhbGwgaWYgbmVlZGVkDQppZiAoIXJlcXVpcmUoZGF0YS50YWJsZSkpIGluc3RhbGwucGFja2FnZXMoJ2RhdGEudGFibGUnKQ0KaWYgKCFyZXF1aXJlKGdncGxvdDIpKSBpbnN0YWxsLnBhY2thZ2VzKCdnZ3Bsb3QyJykNCmlmICghcmVxdWlyZShnZ3Bsb3QyKSkgaW5zdGFsbC5wYWNrYWdlcygnZ2dwbG90MicpDQoNCmxpYnJhcnkoZGF0YS50YWJsZSkNCmxpYnJhcnkoZ2dwbG90MikNCmxpYnJhcnkocGxvdGx5KQ0KYGBgDQoNCiMjIE1hdGggRnVuZGFtZW50YWxzIG9mIHRoZSBQZXJjZXB0cm9uIEFsZ29yaXRobQ0KSW4gb3JkZXIgdG8gdW5kZXJzdGFuZCB0aGUgUGVyY2VwdHJvbiBhbGdvcml0aG0sIGEgYmFzaWMga25vd2xlZGdlIG9mIHRoZQ0KZm9sbG93aW5nIG1hdGhlbWF0aWNhbCBwcmluY2lwbGVzIG11c3QgZmlyc3QgYmUgdW5kZXJzdG9vZDoNCi0gVmVjdG9ycyAoc3BlY2lmaWNhbGx5LCBob3cgdG8gY2FsY3VsYXRlIHRoZSBkaXJlY3Rpb24gYW5kIG5vcm0pDQotIFRoZSBEb3QgUHJvZHVjdA0KDQpJIGFtIGdvaW5nIHRvIGFzc3VtZSB0aGF0IG1ham9yaXR5IG9mIHBlb3BsZSByZWFkaW5nIHRoaXMgYXJ0aWNsZSBoYXZlIHByaW9yIA0Ka25vd2xlZGdlIG9mIGVhY2ggb2YgdGhlc2UgZnVuZGFtZW50YWxzLiBJZiB5b3UgZG8gbm90LCBhIGdyZWF0IHJlc291cmNlIGNhbiBiZQ0KZm91bmQgW2hlcmVdKGh0dHBzOi8vd3d3LmtoYW5hY2FkZW15Lm9yZy9tYXRoL3ByZWNhbGN1bHVzL3g5ZTgxYTRmOTgzODllZmRmOnZlY3RvcnMpLiBJIHdpbGwgYWxzbyBnaXZlIGEgYnJpZWYgcmVmcmVzaGVyIHRvIHByb3ZpZGUgc29tZSBjb250ZXh0Lg0KDQojIyMgVmVjdG9ycw0KDQpgYGB7cn0NCg0KYGBgDQoNCiMjIyBUaGUgRG90IFByb2R1Y3QNCg0KYGBge3J9DQoNCmBgYA0KDQojIyMgTGluZWFyIFNlcGFyYWJsZSBEYXRhDQpUaGUgZGVmaW5pdGlvbiBvZiBsaW5lYXJseSBzZXBhcmFibGUgZGF0YSBpcyBwcmV0dHkgc2VsZi1leHBsYW5hdG9yeS4gQXQgYSBoaWdoDQpsZXZlbCwgZm9yIGRhdGEgdG8gYmUgbGluZWFybHkgc2VwYXJhYmxlIGl0IHNpbXBseSBtZWFucyB0aGF0IHRoZXJlIGV4aXN0cyBzb21lDQpwbGFuZSB0aGF0IGNhbiBsaXRlcmFsbHkgInNlcGFyYXRlIiB0aGUgZGF0YS4gRm9yIGV4YW1wbGUsIGNvbnNpZGVyIHRoZQ0KZm9sbG93aW5nIHBsb3Q6DQoNCmBgYHtyfQ0KdHdvX2Rfc2FtcGxlIDwtIGRhdGEudGFibGUoDQogIHggPSBjKHNlcSgxLCA0LCAuMSksIHNlcSg3LCAxMCwgLjEpKSwNCiAgeSA9IGMocnVuaWYoMzEsIDEsIDQpLCBydW5pZigzMSwgNywgMTApKSwNCiAgY2xhc3NfID0gYXMuZmFjdG9yKGMocmVwKDEsIDMxKSwgcmVwKDAsIDMxKSkpDQopDQoNCnBsdDEgPC0gZ2dwbG90KHR3b19kX3NhbXBsZSwgYWVzKHg9eCwgeT15LCBjb2xvcj1jbGFzc18pKSArIA0KICAgIGdlb21fcG9pbnQoKQ0KDQpwbHQxDQpgYGANCg0KSXQgaXMgcHJldHR5IG9idmlvdXMgdGhhdCB5b3UgY2FuIGRyYXcgc29tZSBhcmJpdHJhcnkgbGluZSB0aGF0IHNlcGFyYXRlcyB0aGUgDQp0d28gZGlmZmVyZW50IGNvbG9yZWQgZGF0YSBwb2ludHM6DQoNCmBgYHtyfQ0KcGx0MSArIGdlb21fYWJsaW5lKHNsb3BlPS0xLjI1LCBpbnRlcmNlcHQ9MTIpDQpgYGANCg0KU2ltaWxhcmx5LCBjb25zaWRlciB0aGUgZm9sbG93aW5nIHRocmVlLWRpbWVuc2lvbmFsIGRhdGE6DQoNCmBgYHtyfQ0KdGhyZWVfZF9zYW1wbGUgPC0gdHdvX2Rfc2FtcGxlWywgeiA6PSBjKHJ1bmlmKDMxLCAxLCA0KSwgcnVuaWYoMzEsIDcsIDEwKSldDQoNCnNjZW5lID0gbGlzdCgNCiAgY2FtZXJhID0gbGlzdCgNCiAgICBjZW50ZXIgPSBsaXN0KHggPSAwLCB5ID0gMCwgeiA9IDApLCANCiAgICBleWUgPSBsaXN0KHggPSAxLjc1LCB5ID0gLjc1LCB6ID0gLjI1KQ0KICAgICkNCiAgKQ0KcGx0MiA8LSBwbG90X2x5KGRhdGE9dGhyZWVfZF9zYW1wbGUsIA0KICAgICAgICAgICAgICAgIHg9fngsIHk9fnksIHo9fnosIA0KICAgICAgICAgICAgICAgIGNvbG9yPX5jbGFzc18sDQogICAgICAgICAgICAgICAgdHlwZT0nc2NhdHRlcjNkJywNCiAgICAgICAgICAgICAgICBtb2RlPSdtYXJrZXJzJywgDQogICAgICAgICAgICAgICAgY29sb3JzPWMoJ2RhcmtyZWQnLCAnYmx1ZScpKSAlPiUNCiAgbGF5b3V0KHNjZW5lPXNjZW5lKQ0KDQpwbHQyDQpgYGANCg0KSW4gdGhpcyBpbnN0YW5jZSwgYSBsaW5lIHdvdWxkIGJlIGluc3VmZmljaWVudCB3aGVuIGF0dGVtcHRpbmcgdG8gc2VwZXJhdGUgdGhlIA0KdHdvIGRhdGEgY2xhc3Nlcy4gSW5zdGVhZCwgYSB0d28tZGltZW5zaW9uYWwgcGxhbmUgY2FuIGJlIHVzZWQ6DQoNCmBgYHtyfQ0KDQpwbHQzIDwtIHBsb3RfbHkoZGF0YT10aHJlZV9kX3NhbXBsZSwgDQogICAgICAgICAgICAgICAgY29sb3JzPWMoJ2RhcmtyZWQnLCAnZ3JheScsICdibHVlJykpICU+JQ0KICBhZGRfdHJhY2UoeD1+c2VxKG1pbih4KSwgbWF4KHgpLCAuMSkqMS41LCANCiAgICAgICAgICAgIHk9fnNlcShtaW4oeSksIG1heCh5KSwgLjEpKjEuNSwgDQogICAgICAgICAgICB6PX5tYXRyaXgoNiwgbnJvdz02MiwgbmNvbD02MiksDQogICAgICAgICAgICB0eXBlPSdzdXJmYWNlJykgJT4lDQogIGFkZF90cmFjZSh4PX54LCB5PX55LCB6PX56LCANCiAgICAgICAgICAgICAgICBjb2xvcj1+Y2xhc3NfLA0KICAgICAgICAgICAgICAgIHR5cGU9J3NjYXR0ZXIzZCcsDQogICAgICAgICAgICAgICAgbW9kZT0nbWFya2VycycpICU+JQ0KICBsYXlvdXQoc2NlbmU9YXBwZW5kKHNjZW5lLCANCiAgICAgICAgICAgICAgICAgICAgICBsaXN0KHhheGlzPWxpc3QodGl0bGU9J3gnKSwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgIHlheGlzPWxpc3QodGl0bGU9J3knKSwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgIHpheGlzPWxpc3QodGl0bGU9J3onKSkpKSAlPiUgDQogIGhpZGVfY29sb3JiYXIoKQ0KDQpwbHQzDQpgYGANCg0KT2YgY291cnNlLCB0aGlzIGdldHMgYSBsaXR0bGUgaGFyZGVyIHRvIHZpc3VhbGl6ZSAtIG9yIGV2ZW4gZXhwbGFpbiAtIA0KYXMgZGltZW5zaW9uYWxpdHkgY29udGludWVzIHRvIGluY3JlYXNlLiBBcyBhIHJlc3VsdCwgYSBtb3JlIGdlbmVyYWxpemVkIHRlcm0NCnRvIGRlc2NyaWJlIHRoaXMgbGluZWFyIHNlcGFyYXRpb24gaXMgdXNlZDogYSAqKmh5cGVycGxhbmUqKi4NCg0KIyMjIEh5cGVycGxhbmVzDQpBY2NvcmRpbmcgdG8gdGhlIFtXaWtpcGVkaWEgZGVmaW5pdGlvbl0oaHR0cHM6Ly9lbi53aWtpcGVkaWEub3JnL3dpa2kvSHlwZXJwbGFuZSk6IA0KIkluIGdlb21ldHJ5LCBhIGh5cGVycGxhbmUgaXMgYSBzdWJzcGFjZSB3aG9zZSBkaW1lbnNpb24gaXMgb25lIGxlc3MgdGhhbiANCnRoYXQgb2YgaXRzIGFtYmllbnQgc3BhY2UuIiBUaGlzIG1ha2VzIHNlbnNlLiBMb29raW5nIGF0IG91ciBmaXJzdCBleGFtcGxlLCB0aGUNCmRlbWluc2lvbmFsaXR5IG9mIHRoZSBkYXRhIGlzIHR3bywgYW5kIHRoZSBsaW5lIGRyYXduIHRvIHNlcGFyYXRlIHRoZSBkYXRhIGlzDQpvbmUtZGltZW5zaW9uYWwuIEluIHRoZSBzZWNvbmQgZXhhbXBsZSwgdGhlIGRhdGEgaXMgdGhyZWUtZGltZW5zaW9uYWwgYW5kIHRoZQ0Kc2VwYXJhdGlvbiBpcyB0d28uDQoNCiMjIFBlcmNlcHRyb24gQWxnb3JpdGhtDQoNCiMjIyBHZW5lcmF0ZSBEYXRhDQoNCmBgYHtyfQ0KDQpgYGANCg0KIyMjIFdlaWdodHMNCg0KYGBge3J9DQoNCmBgYA0KDQojIyMgSXRlcmF0ZQ0KDQpgYGB7cn0NCg0KYGBgDQoNCg0KIyMgTXVsdGktTGF5ZXIgUGVyY2VwdHJvbg0KDQpgYGB7cn0NCg0KYGBgDQoNCg0KDQoNCg0KDQoNCg==